Hearing and speech impaired electronic device control

ABSTRACT

A sign language message may be derived from user movements detected proximate to an electronic device by comparing the movements to a database that includes data regarding one or more sign languages. A control function may then be identified which the electronic device may be caused to perform. Output related to the user&#39;s detected movements may be provided to the user. In some implementations, output specifying identified control functions may be transmitted to a presentation device. Further, in some implementations, if a sign language message and/or a control function cannot be unambiguously derived and/or identified, output promoting the user to provide additional information may be transmitted to a presentation device. Detected movements may be compared to data regarding multiple different sign languages and/or a subset of available multiple different sign languages.

FIELD OF THE INVENTION

This disclosure relates generally to control of electronic devices, andmore specifically to utilizing sign language to control electronicdevices.

SUMMARY

The present disclosure discloses systems and methods for hearing and/orspeech impaired control of electronic devices. A sign language messagemay be derived from movements of a user detected proximate to anelectronic device utilizing a movement sensor by comparing the movementsto a database that includes data regarding one or more sign languages. Acontrol function of the electronic device may then be identified basedon analysis of the sign language message and the electronic device maybe caused to perform the control function. As such, the user is able toutilize sign language to control the electronic device.

Output related to the user's detected movements may be provided to theuser. In some implementations, output specifying identified controlfunctions may be transmitted to a presentation device. Further, in someimplementations, if a sign language message cannot be unambiguouslyderived from the user's movements and/or a control function cannot beunambiguously identified from a sign language message, output promotingthe user to provide additional information may be transmitted to apresentation device. The ambiguity may then be resolved based at leaston the addition information provided by the user in response to theprompt.

Detected movements may be compared to data regarding multiple differentsign languages. However, in various implementations detected movementsmay be compared to data regarding a particular sign language. In some ofthese various implementations, the detected movements may be compared todata regarding a particular sign language based on a setting, such as auser preference or a default setting. However, in other of these variousimplementations, the particular sign language of a sign language messagemay be determined by comparing a first set of movements to dataregarding multiple different sign languages and deriving a first signlanguage message component. Then, once the particular sign language isidentified, subsequent sets of movements may be compared to just dataregarding the determined particular sign language to derive additionalsign language message components.

It is to be understood that both the foregoing general description andthe following detailed description are for purposes of example andexplanation and do not necessarily limit the present disclosure. Theaccompanying drawings, which are incorporated in and constitute a part'of the specification, illustrate subject matter of the disclosure.Together, the descriptions and the drawings serve to explain theprinciples of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system for hearing and/orspeech impaired control of an electronic device.

FIG. 2 is a flow chart illustrating a method for hearing and/or speechimpaired control of an electronic device. This method may be performedby the system of FIG. 1.

FIGS. 3A-3E are diagrams illustrating a user utilizing a system forcontrolling an electronic device. The system may be the system of FIG.1.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The description that follows includes sample systems, methods, andcomputer program products that embody various elements of the presentdisclosure. However, it should be understood that the describeddisclosure may be practiced in a variety of forms in addition to thosedescribed herein.

Electronic devices (such as computers, telephones, microwave ovens andother kitchen appliances, televisions, lighting and climate systems,radios, and so on) abound in the modern world. As electronic devicesbecome increasingly present in the lives of users, manufacturers mayattempt to provide increasingly simplistic and convenient ways for usersto control such electronic devices. Users may be able to control someelectronic devices utilizing speech. In other words, the user may beable to speak commands to an electronic device and the electronic devicemay understood and perform the spoken commands. By way of a firstexample, a user may tell a set top box to switch to a desired channel.By way of a second example, a user may tell an oven to activate abroiler. Additionally, some devices may be capable of outputting speechmessages, essentially “talking” to a user. For example, a microwave mayplay a speech message (such as a pre-recorded speech message, asynthesized speech message, and so on) requesting that a user turn overan item being defrosted.

However, speech impaired users may not be able to control electronicdevices that are capable of being controlled by speech. Further, hearingimpaired users may not be able to recognize speech output provided by anelectronic device. As such, users with speech and/or hearing impairmentsmay not be able to take advantage of the simpler and more convenientspeech interfaces of various electronic devices. Instead, such users maybe required to utilize less convenient physical interfaces, such askeyboards, buttons, touch screens, and so on.

Moreover, even users who do not have hearing and/or speech impairmentsmay not always be able to talk to electronic devices or hear theirelectronic devices “talk” back. For example, when utilizing anelectronic device in a high noise environment such as a rock concert ora quiet required environment such as a library, even users withouthearing and/or speech impairments may be unable to use speechrecognition and/or speech output functions of the electronic device.Thus, in such an environment, the user may be required to utilize lessconvenient physical interfaces, such as keyboards, buttons, touchscreens, and so on, even though the user is not hearing and/or speechimpaired.

The present disclosure discloses systems and methods for hearing and/orspeech impaired control of electronic devices. Movements of a userlocated proximate to an electronic device may be detected. A signlanguage message may be derived from the detected movements by comparingthe movements to a database of one or more sign languages. The signlanguage message may then be analyzed to identify a control function ofthe electronic device that is associated with the sign language message.Thus, a user may utilize sign language to control the electronic device.

FIG. 1 is a block diagram illustrating a system 100 for hearing and/orspeech impaired control of an electronic device. The system 100 includesa electronic device 101 which may be any kind of electronic device (suchas a computer, a telephone, a microwave or other kitchen appliance, atelevision, a set top box, a lighting and/or climate system, a radio, adigital music player, a digital video recorder, a digital video discplayer, and so on) and may perform a number of different controlfunctions in response to user instruction.

The electronic device may include one or more processing units 102, oneor more movement sensors 103, and one or more non-transitory storagemedia 104 (which may take the form of, but is not limited to, a magneticstorage medium; optical storage medium; magneto-optical storage medium;read only memory; random access memory; erasable programmable memory;flash memory; and so on). The storage medium may store a movementdatabase 105 that includes movements that correspond to sign languagemessage components one or more sign languages (such as American SignLanguage, Russian Sign Language, Ukrainian Sign Language, Chinese SignLanguage, British Sign Language, International Sign, and so on). Themovement sensor may be any kind of sensor that is operable to detectmovement including, but not limited to, a camera (such as a video cameraor a still image camera), an infrared motion detector, an ultrasonicmotion detector, a microwave sensor, and so on. Additionally, in someimplementations the movement sensor may be incorporated into anapparatus (such as gloves or other items of clothing) that may be wornby a user and includes positioning sensors (such as global positioningsensors) for detecting movement of the wearer.

The processing unit 102 may execute instructions stored in the storagemedium 104 to detect user movements utilizing the movement sensor 103,compare the detected movements to the movement database 105 to derive asign language message from the detected movements, and identify acontrol function which the electronic device 101 is operable to performthat is associated with the sign language message. After the processingunit identifies the control function, the processing device may causethe electronic device to perform the control function, such as byexecuting instructions stored in the storage medium that are associatedwith the identified control function.

As sign languages may include multiple different ways of phrasing thesame statement, the processing unit 102 may identify a single associatedcontrol function from multiple different sign language messages. Forexample, a set top box may identify a “switch channel to two-hundredtwenty-two” control function from sign language messages that includesign language message components “two two two,” “two hundred twentytwo,” two hundred and twenty two,” and so on.

The system 100 may also include one or more presentation devices 107(such as a display screen, a speaker, and so on) and the electronicdevice 101 may include one or more input/output components 106 fortransmitting output to the presentation device. The processing unit 102may transmit an output message to the presentation device via theinput/output component after deriving a sign language message from thedetected movement. The processing unit may transmit the output messageto notify the user as to the sign language message that was derived fromthe detected movements, the control function that was identified asrelated to the derived sign language message, and so on. The outputmessage may include a variety of presentation forms such as text todisplay, recorded speech, synthesized speech, images or video of signlanguage, combinations thereof, and so on.

The processing unit 102 may not always be able to unambiguously derive asign language message from the detected movement, identify a controlfunction associated with a derived sign language message, and so on. Insome implementations, when the processing unit determines that it cannotunambiguously derive a sign language message from the detected movement,the processing unit may transmit an output message to the presentationdevice 107 via the input/output component 106 that requests the user tore-perform the sign language related to the attempted sign languagemessage. For example, if the processing unit is not able to derive asign language message from the detected movement because the detectedmovement does not correspond to any movements in the movement database105, the processing unit may transmit an output message to thepresentation device via the input/output component that indicates thatthe sign language was not understood and requests the user tore-perform.

Further, in some implementations, when the processing unit determinesthat it cannot unambiguously identify a control function associated witha derived sign language message, the processing unit 102 may transmit anoutput message to the presentation device 107 via the input/outputcomponent 106 that requests the user provide additional information toclarify the ambiguity. Then, after the processing unit receives theadditional information from the user, the processing unit may identifythe control function from a combination of the identified sign languagemessage and the additional information. For example, the electronicdevice may be a microwave and the processing unit may derive a signlanguage message of “thirty seconds” from detected movement. Themicrowave may include control functions to cook for thirty seconds aswell as defrost for thirty seconds and the processing unit may not beable to determine which control function is intended without additionalinformation. As such, the processing unit may transmit an output messageto the presentation device via the input/output component that promptsthe user to select either the cook control function or the defrostcontrol function.

Although the processing unit 102, movement sensor 103, storage medium104, and input/output component 106 are shown and described above asbeing incorporated into the electronic device 101, in variousimplementations one or more of the processing unit, movement sensor,storage medium and/or input/output component may be separate from theelectronic device and may communicate with the electronic device toperform the above described functions.

Further, although the above describes control of the electronic device101 utilizing fully developed sign languages intended for communication,in various implementations the sign languages may include pantomimecommunication systems specifically developed for controlling theelectronic device or similar electronic devices that consist of a fewsigns relating to control functions. For example, a set top box mayinclude pre-defined functionality to recognize a system of pantomimescorresponding to channels to which to switch the set top box thatconsists of the user holding up a number of fingers to indicate channelnumbers in sequence. By way of a second example, instead of includingpre-defined functionality to recognize the system of pantomimes, a settop box may include the ability to record user pantomimes to associatewith control functions which may then be detected and recognized.

FIG. 2 illustrates a method 200 for hearing and/or speech impairedcontrol of an electronic device. The method 200 may be performed by theelectronic device 101 of FIG. 1. The flow begins at block 201 andproceeds to block 202 where the electronic device begins operations. Theflow then proceeds to block 203 where the processing unit 102 determineswhether or not movement is detected by the movement sensor 103. If so,the flow proceeds to block 204. Otherwise, the flow returns to block 202where the electronic device continues operating.

At block 204, the processing unit 102 compares the detected movement tothe movement database 105. The flow then proceeds to block 205 where theprocessing unit derives a sign language message from the detectedmovement based on the comparison of the detected movement to themovement database. Next, the flow proceeds to block 206 where theprocessing unit determines whether or not additional information isneeded to identify a control function associated with the sign languagemessage. If so, the flow proceeds to block 211. Otherwise, the flowproceeds to block 207.

At block 207, after the processing unit 102 determines that additionalinformation is not needed, the processing unit identifies the controlfunction associated with the sign language message. The flow thenproceeds to block 208 where the processing unit causes the electronicdevice 101 to perform the control function. Next, the flow proceeds toblock 209 processing unit determines whether or not to output theidentified control function. If so, the flow proceeds to block 210.Otherwise, the flow returns to block 202 where the electronic devicecontinues operating. The processing unit may base the determination ofwhether or not to output the identified control function based on one ormore settings, such as user preferences, stored in the storage medium104.

At block 210, after the processing unit 102 determines to output theidentified control function, the processing unit transmits an outputmessage which specifies the identified control function to thepresentation device 107 via the input/output component 106. The flowthen returns to block 202 where the electronic device continuesoperating.

At block 211, after the processing unit 102 determines that additionalinformation is needed to identify the control function associated withthe sign language message, the processing unit prompts a user foradditional information by transmitting an output message to thepresentation device 107 via the input/output component 106. The flowthen proceeds to block 212 where the processing unit receives theadditional information from the user. Next, the flow proceeds to block213 where the processing unit identifies the control function signlanguage message based on the sign language message and the additionalinformation. The flow then proceeds to block 208 where the processingunit causes the electronic device 101 to perform the control function.

Returning to FIG. 1, in some implementations, when the processing unit102 compares detected movements to the movement database 105, theprocessing unit may compare the detected movements to data regardingmultiple different sign languages stored in the movement database.

However, in other implementations the processing unit 102 may comparethe detected movements to data regarding a particular sign languagestored in the movement database 105 instead of all of the different signlanguages stored in the movement database. In this way, less time and/orresources may be required for the comparison. For example, the movementdatabase may include data regarding movements of American Sign Language,Russian Sign Language, and Ukrainian Sign Language. However, theprocessing unit may compare detected movements to just data regardingmovements of American Sign Language. The processing unit may comparedetected movements to data regarding a particular sign language based ona setting, such as a user preference or a default setting, stored in thestorage medium 104.

In still other implementations, the processing unit 102 may determinethe particular sign language of a sign language message by comparing afirst set of the detected movements to data regarding multiple differentsign languages stored in the movement database 105. Based on thecomparison of the first set of movements, the processing unit mayidentify a first sign language message component of the sign languagemessage. The processing unit may then compare subsequent sets of thedetected movements to just data regarding the determined particular signlanguage to identify additional sign language message components.However, if the processing unit then determines based on comparison tothe data regarding the determined particular sign language that asubsequent set of the detected movements does not correspond to theparticular sign language, the processing unit may compare the subsequentset of the detected movements to data regarding multiple different signlanguages stored in the movement database.

For example, the movement database 105 may include data regardingmovements of American Sign Language, Chinese Sign Language, andUkrainian Sign Language. The processing unit 102 may compare a first setof the detected movements to data regarding American Sign Language,Chinese Sign Language, and Ukrainian Sign Language and identify that thesign language message is in Chinese Sign Language. The processing unitmay then compare subsequent portions of the detected movements to justdata regarding Chinese Sign Language. However, if the processing unitthen determines based on comparison to the data regarding Chinese signlanguage that a subsequent set of the detected movements does notcorrespond to Chinese sign language, the processing unit may compare thesubsequent set of the detected movements to data regarding American SignLanguage, Chinese Sign Language, and Ukrainian Sign Language.

FIGS. 3A-3E are illustrate a user 303A-303E utilizing a system 300A-300Efor controlling an electronic device 301A-301E. The system may be thesystem of FIG. 1. FIG. 3A depicts a user 303A watching a display screen304A of a television 301A. As illustrated, the television is displayingprogramming corresponding to a channel forty-two on the display screen.The television includes a video camera 302A that detects movements ofthe user.

FIG. 3B depicts changes to FIG. 3A after the television 301A detectsthat the user 303A performed sign language with a message “switch tochannel two hundred and forty two” where the television is notconfigures(to output control functions identified from sign languagesignals. As illustrated, the television 301 B displays programmingcorresponding to a channel two forty-two on the display screen 304B.

FIG. 3C depicts changes to FIG. 3A after the television 301A detectsthat the user 303A performed sign language with a message “switch tochannel two hundred and forty two” where the television is configured tooutput control functions identified from sign language signals. Asillustrated, the television 301C displays an output message on displayscreen 304C indicating that a command to switch the channel to channeltwo hundred forty-two has been received. Then, the television willswitch the channel to display programming corresponding to a channel twoforty-two on the display screen, as illustrated in FIG. 3B.

FIG. 3D depicts changes to FIG. 3A after the television 301A detectsthat the user 303A performed sign language with a message “up.” Asillustrated the television 301D is unable to unambiguously determinewhether the user desires a channel up function or a volume up functionand displays an output message on the display screen 304D that promptsthe user 303D for additional information. FIG. 3E depicts changes toFIG. 3D after the television 301 D detects that the user 303D performedsign language with a message “channel” in order to provide theadditional information requested. As illustrated, the television 301Edisplays programming corresponding to a channel forty-three on thedisplay screen 304E.

In the present disclosure; the methods disclosed may be implemented assets of instructions or software readable by a device. Further, it isunderstood that the specific order or hierarchy of steps in the methodsdisclosed are examples of sample approaches. In other embodiments, thespecific order or hierarchy of steps in the method can be rearrangedwhile remaining within the disclosed subject matter. The accompanyingmethod claims present elements of the various steps in a sample order,and are not necessarily meant to be limited to the specific order orhierarchy presented.

The described disclosure may be provided as a computer program product,or software, that may include a non-transitory machine-readable mediumhaving stored thereon instructions, which may be used to program acomputer system (or other electronic devices) to perform a processaccording to the present disclosure. A non-transitory machine-readablemedium includes any mechanism for storing information in a form (e.g.,software, processing application) readable by a machine (e.g., acomputer). The non-transitory machine-readable medium may take the formof, but is not limited to, a magnetic storage medium (e.g., floppydiskette, video cassette, and so on); optical storage medium (e.g.,CD-ROM); magneto-optical storage medium; read only memory (ROM); randomaccess memory (RAM); erasable programmable memory (e.g., EPROM andEEPROM); flash memory; and so on.

It is believed that the present disclosure and many of its attendantadvantages will be understood by the foregoing description, and it willbe apparent that various changes may be made in the form, constructionand arrangement of the components without departing from the disclosedsubject matter or without sacrificing all of its material advantages:The form described is merely explanatory, and it is the intention of thefollowing claims to encompass and include such changes.

While the present disclosure has been described with reference tovarious embodiments, it will be understood that these embodiments areillustrative and that the scope of the disclosure is not limited tothem. Many variations, modifications, additions, and improvements arepossible. More generally, embodiments in accordance with the presentdisclosure have been described in the context or particular embodiments.Functionality may be separated or combined in blocks differently invarious embodiments of the disclosure or described with differentterminology. These and other variations, modifications, additions, andimprovements may fall within the scope of the disclosure as defined inthe claims that follow.

The invention claimed is:
 1. A method for controlling an electronicdevice using sign language, the method comprising: detecting a pluralityof user movements associated with a user utilizing at least one movementsensor; deriving, utilizing at least one processing unit, a first signlanguage message corresponding to control of an electronic device fromat least one user movement of the plurality of user movements;identifying, utilizing the at least one processing unit, a controlfunction of the electronic device associated with the first signlanguage message; identifying a conflict wherein a first controlfunction and a second control function of the electronic device are eachdetermined to be candidate control functions that are associated withthe first sign language message; transmitting, in response toidentifying the conflict, a notification to the user via a displaydevice associated with the electronic device, wherein the notificationincludes a request for the user to repeat the plurality of usermovements due to the conflict; detecting, after transmitting thenotification to the user, a second plurality of user movementsassociated with the user utilizing the movement sensor; deriving asecond sign language message corresponding to control of the electronicdevice, wherein the second sign language message is derived from atleast one user movement of the second plurality of user movements;identifying a control function of the electronic device associated withthe second sign language message; resolving the conflict to determinethat one of the first control function and the second control functionis associated with the first sign language message using the secondplurality of user movements and the second sign language message; andimplementing one of the first control function and the second controlfunction based on the determination.
 2. The method of claim 1, furthercomprising transmitting an output message to an output device inresponse to the first sign language message.
 3. The method of claim 2,wherein the output message comprises at least one of a text message, asign language message, and an audio message.
 4. The method of claim 1,wherein said operation of deriving, utilizing at least one processingunit, a first sign language message corresponding to control of anelectronic device from at least one user movement of the plurality ofuser movements comprises identifying at least one sign language messagecomponent by comparing the plurality of user movements to a database ofmovements associated with movements of one or more sign languages. 5.The method of claim 4, wherein said operation of comparing the pluralityof user movements to a database of movements associated with movementsof one or more sign languages further comprises comparing the pluralityof user movements to movements of a particular sign language stored inthe database based on a setting specified by a user.
 6. The method ofclaim 4, wherein said operation of comparing the plurality of usermovements to a database of movements associated with movements of one ormore sign languages further comprises: determining a particular signlanguage of the sign language message by comparing a first set of usermovements of the plurality of user movements to movements of a pluralityof sign languages stored in the database to identify a first signlanguage message component; and comparing at least a second set of usermovements of the plurality of user movements to a subset of the databaseto identify at least a second sign language message component whereinthe subset comprises movements of the particular sign language.
 7. Themethod of claim 6, further comprising: determining that at least oneadditional set of user movement of the plurality of user movements doesnot correspond to the particular sign language; and identifying at leastan additional sign language message component by comparing the at leastone additional set of user movement of the plurality of user movementsto movements of the plurality of sign languages stored in the database.8. The method of claim 1, further comprising: transmitting a query tothe user, wherein the query includes a request for the user to choose anintended option of a set of predetermined options, wherein the set ofpredetermined options include the first control function and the secondcontrol function; and receiving a response to the query from the user,wherein the response includes an indication that an option of the set ofpredetermined options is the intended option of the user.
 9. The methodof claim 8, further comprising: comparing the response to the queryreceived from the user to the first control function and the secondcontrol function, wherein the comparison is used to determine if theconflict was resolved correctly.
 10. The method of claim 8, furthercomprising: in response to receiving the response to the query from theuser, determining that the response was received before resolving theconflict; and implementing one of the first control function and thesecond control function based on the response to the query instead ofthe determination.
 11. The method of claim 1, further comprising:detecting one or more pantomime user movements; receiving one or moreinputs from the user, wherein the one or more inputs are each associatedwith one or more of the detected one or more pantomime user movements;and storing the pantomime user movements and their associated userinputs.
 12. The method of claim 11, wherein the plurality of usermovements include pantomime user movements, and wherein the first signlanguage messages are obtained from the stored pantomime user movementsand their associated user inputs.
 13. A system for controlling anelectronic device, comprising: an electronic device; at least onemovement sensor that detects a plurality of user movements associatedwith a user; at least one non-transitory storage medium that stores adatabase of movements associated with movements of one or more signlanguages; and at least one processing unit; wherein the at least oneprocessing unit executes instructions stored in the at least onenon-transitory storage medium to derive a first sign language messagecorresponding to control of the electronic device from at least one usermovement of the plurality of user movements, identify a conflict whereina first control function and a second control function of the electronicdevice are each determined to be a candidate control function that isassociated with the first sign language message, transmit, in responseto identifying the conflict, a notification to the user via a displaydevice associated with the electronic device, wherein the notificationincludes a request for the user to repeat the plurality of usermovements due to the conflict, detect, after transmitting thenotification to the user, a second plurality of user movementsassociated with the user utilizing the movement sensor, derive a secondsign language message corresponding to control of the electronic device,wherein the second sign language message is derived from at least oneuser movement of the second plurality of user movements, identify acontrol function of the electronic device associated with the secondsign language message; resolve the conflict to determine that one of thefirst control function and the second control function is associatedwith the first sign language message using the second plurality of usermovements and the second sign language message, and implement one of thefirst control function and the second control function based on thedetermination.
 14. The system of claim 13, wherein the at least onemovement sensor comprises at least one of a video camera, a still imagecamera, an infrared sensor, an ultrasonic sensor, and a microwavesensor.
 15. The system of claim 13, wherein the at least one movementsensor is incorporated into the electronic device.
 16. The system ofclaim 13, wherein the at least one processing unit: determines aparticular sign language of a sign language message by comparing a firstportion of a plurality of user movements to movements of a plurality ofsign languages stored in the database to identify a first sign languagemessage component; and compares at least a second portion of a pluralityof user movements to a subset of the database to identify at least asecond sign language message component wherein the subset comprisesmovements of the particular sign language.
 17. The system of claim 13,wherein the at least one processing unit derives a sign language messagefrom the at least one user movement of the plurality of user movementsby comparing the plurality of user movements to movements of aparticular sign language stored in the database based on a settingspecified by a user.
 18. The system of claim 13, wherein the at leastone processing unit transmits a control function message to an outputdevice indicating the identified control function associated with thesecond sign language message.
 19. The system of claim 13, wherein the atleast one processing unit: transmits a query to the user, wherein thequery includes a request for the user to choose an intended option of aset of predetermined options, wherein the set of predetermined optionsinclude the first control function and the second control function; andreceives a response to the query from the user, wherein the responseincludes an indication that an option of the set of predeterminedoptions is the intended option of the user.
 20. The system of claim 13,wherein the at least one processing unit: detects one or more pantomimeuser movements; receives one or more inputs from the user, wherein theone or more inputs are each associated with one or more of the detectedone or more pantomime user movements; and stores, in the at least onenon-transitory storage medium, the pantomime user movements and theirassociated user inputs.